Model Inference with vLLM

Overview

This guide serves an open-weight model from Hugging Face on a GPU, using vLLM’s OpenAI-compatible server. You deploy the upstream vllm/vllm-openai image as a Scaling Group and set its entrypoint to vLLM’s API server. Chalk supplies the GPU, the container, and the URL. vLLM already ships the HTTP server, so you do not build an image or write serving code.

The example serves Qwen3 4B Instruct, which anyone can download without a Hugging Face account. Every flag after api_server in the entrypoint is vLLM’s, so you can serve another model by changing --model.

To serve a model behind Chalk features and queries, see Model Deployments.

Before you begin

You need a GPU in your cluster. The example requests one NVIDIA T4. The GPU types available to you depend on the node pools in your cluster. See Resources and GPUs.

Deploy the model

Create deploy_vllm.py:

import chalkcompute

sg = chalkcompute.ScalingGroup(
    image="vllm/vllm-openai:latest",
    name="qwen-vllm",
    gpu="nvidia-tesla-t4:1",
    cpu="4",
    memory="16Gi",
    port=8000,
    entrypoint=[
        "python3", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "Qwen/Qwen3-4B-Instruct-2507",
        "--port", "8000",
        "--max-model-len", "4096",
        "--dtype", "half",
        "--gpu-memory-utilization", "0.90",
    ],
    readiness_probe=chalkcompute.ReadinessProbe.http("/health"),
    authenticated=True,
    min_replicas=1,
    max_replicas=2,
)

sg.deploy(ready_timeout=1800.0)
print(sg.web_url)

Run it:

$ python deploy_vllm.py

deploy() returns when a replica is ready and prints the group’s URL. The readiness probe reports a replica as ready once vLLM’s /health endpoint responds, which happens after the model weights are loaded onto the GPU. The first deploy can take 20 minutes or more, while a GPU node starts and vLLM downloads and loads the weights, so ready_timeout is raised from its 300-second default to 30 minutes.

Parameter	Purpose
`gpu="nvidia-tesla-t4:1"`	Request one T4 GPU for each replica.
`readiness_probe=...`	Send traffic to a replica only after vLLM has loaded the model.
`authenticated=True`	Require a Chalk access token on every request. See Securing the endpoint.
`min_replicas=1`, `max_replicas=2`	Keep one replica running, and add a second under load. See Scaling.

Requesting a GPU

gpu takes a "<type>:<count>" string. The type selects the node pool, and the count sets the number of GPUs each replica gets. "nvidia-tesla-t4:1" gives each replica one T4, which serves a 4B parameter model at 16-bit precision. The T4 does not support bfloat16, so the example sets --dtype half. On an L4 or a newer GPU, you can set --dtype bfloat16 instead.

Without gpu, the group requests no GPU, its replicas run on CPU nodes, and the vLLM image does not start. See Resources and GPUs for the GPU types and for clusters where you pass a count instead of a type.

Serving a different model

Change --model in the entrypoint:

    entrypoint=[
        "python3", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "Qwen/Qwen3-8B",
        "--port", "8000",
        "--max-model-len", "8192",
    ],

Chalk passes the list through as the container’s entrypoint, so every flag vLLM’s server accepts works here. Choose the GPU, --max-model-len, and --gpu-memory-utilization to fit the model. A model too large for the GPU fails at startup with an out-of-memory error while loading weights.

Serving a gated model

Some models, including the Gemma and Llama families, are gated: Hugging Face serves their weights only to accounts that have accepted the model’s license. To serve one:

Accept the license on the model’s Hugging Face page, and create an access token for the same account.
In the Chalk dashboard, go to Integrations → Secrets & Variables and add a secret named HF_TOKEN with the token as its value.

Pass the secret to the group, which makes it available to vLLM:

    secrets=[chalkcompute.Secret.from_chalk_env("HF_TOKEN")],

To use a token from your shell instead of the dashboard, write Secret.from_local_env("HF_TOKEN"), which stores the value as a Chalk secret when you deploy.

If vLLM cannot download a gated model, it exits with an authorization error from huggingface.co and the group never becomes ready. The replica logs show the error.

Query the endpoint

vLLM exposes an OpenAI-compatible API under /v1. Because the group sets authenticated=True, every request needs a Chalk access token. Put the group’s URL and a token in your shell:

$ export WEB_URL=<the URL printed by deploy_vllm.py>
$ export CHALK_TOKEN=$(chalk token get)

The OpenAI client sends its api_key as a bearer token, so pass the Chalk token there:

import os

from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['WEB_URL']}/v1",
    api_key=os.environ["CHALK_TOKEN"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-4B-Instruct-2507",
    messages=[{"role": "user", "content": "Explain feature stores in two sentences."}],
)

print(response.choices[0].message.content)

The model value must match the --model string in the entrypoint, because vLLM serves the model under that name.

With curl:

$ curl "$WEB_URL/v1/chat/completions" \
    -H "Authorization: Bearer $CHALK_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Qwen/Qwen3-4B-Instruct-2507",
      "messages": [{"role": "user", "content": "Hello!"}]
    }'

Scaling

min_replicas and max_replicas set the range the autoscaler works in. Each replica runs on its own GPU.

min_replicas=1 keeps one replica, and its GPU, running while the group is idle. To have the group scale to zero when traffic stops, set min_replicas=0. While the group starts a replica, requests fail unless you set a scale-from-zero request policy. See Autoscaling for scale to zero, the request policy, and GPU utilization targets.

Securing the endpoint

A scaling group’s URL is public by default: the gateway in front of it is the cluster’s internet-facing load balancer, so anyone with the URL can reach your GPU. Protect the group on its first deploy, with a Chalk access token or a vLLM API key.

Private routing, which keeps a group off the public internet, requires setup with Chalk. To use it, contact Chalk support.

Require a Chalk access token

The example sets authenticated=True. The gateway in front of the group then requires a Chalk access token on every request, on every path, before the request reaches vLLM. See Authentication for how tokens work.

Set a vLLM API key

vLLM’s server reads VLLM_API_KEY from its environment and requires it as a bearer token. To use a key instead of a Chalk access token, remove authenticated=True and add the key to secrets:

    secrets=[chalkcompute.Secret.from_chalk_env("VLLM_API_KEY")],

Callers then send Authorization: Bearer $VLLM_API_KEY, and the OpenAI client takes the key as its api_key.

The key protects only these path prefixes: /v1, /v2, /inference, and /cohere. Other paths do not require it. That includes POST /invocations, which runs the same inference as /v1/chat/completions. vLLM’s security guidance says: “Do not rely exclusively on --api-key for securing access to vLLM.”

A request carries one Authorization header, which holds either a Chalk access token or a vLLM key. Set authenticated=True or VLLM_API_KEY, not both.

​Overview

​Before you begin

​Deploy the model

​Requesting a GPU

​Serving a different model

​Serving a gated model

​Query the endpoint

​Scaling

​Securing the endpoint

​Require a Chalk access token

​Set a vLLM API key

On this page